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ABSTRACT 


An  optimal  convergence  condition  for  Newton  iteration  in  a Banach  space 
is  established.  There  exist  problems  for  which  the  iteration  converges  but 
the  complexity  is  unbounded.  We  show  what  stronger  condition  must  be  imposed 
to  also  assure  "good  complexity". 
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1.  INTRODUCTION 

Numerous  papers  have  analyzed  sufficient  conditions  for  the  convergence 
of  algorithms  for  the  solution  of  non-linear  problems.  In  addition  to  con- 
vergence, we  consider  another  fundamental  question.  What  stronger  conditions 
must  be  imposed  to  assure  "good  complexity"?  This  is  clearly  one  of  the 
crucial  issues  (in  addition  to  stability)  if  one  is  interested  in  actual  com- 
putation. We  believe  it  is  also  a most  interesting  theoretical  question. 

We  consider  Newton  iteration  for  a simple  zero  of  a non-linear  operator 
in  a Banach  space  of  finite  or  infinite  dimension.  We  establish  the  optimal 
radius  of  the  ball  of  convergence  with  respect  to  a certain  functional.  There 
exist  problems  where  the  iteration  converges  but  the  complexity  increases  log- 
arithmically to  infinity  as  the  initial  iterate  approaches  the  boundary  of 
the  ball  of  convergence.  (This  phenomenon  does  not  occur  in  the  Kantorovich 
theory  of  operator  equations;  see  Section  3.)  We  establish  the  optimal  radius 
of  the  ball  of  good  complexity. 

In  this  paper  we  limit  ourselves  to  the  important  case  of  Newton  iteration. 
In  other  papers  (Traub  and  Wozniakowski  [76b]  and  [77])  we  study  optimal  con- 
vergence and  complexity  for  classes  of  iterations. 

We  summarize  the  results  of  this  paper.  Definitions  and  theorems  con- 
cerning the  optimal  ball  of  convergence  are  given  in  Section  2.  We  conclude 
this  Section  by  giving  conditions  under  which  the  radius  of  the  ball  of  con- 
vergence is  a constant  fraction  of  the  radius  of  the  ball  of  analytic ity  of 
the  operator. 

Complexity  of  Newton  iteration  is  studied  in  Section  3.  We  show  that 


Newton  iteration  may  converge  but  have  arbitrarily  high  complexity  and  conjec- 
ture this  is  a general  phenomenon.  We  establish  the  radius  of  the  ball  of 
good  complexity  as  well  as  a lower  bound  on  the  complexity  of  Newton  iteration. 
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2.  CONVERGENCE  OF  NEWTON  ITERATION 

We  consider  the  solution  of  the  non-linear  equation 
(2.1)  F (x)  - 0 


where  F:  D c and  B^,  B2  are  Banach  spaces  over  the  real  or  complex 

fields  of  dimension  N,  N = dim(B^)  “ dim(B2) , 1 5 N < +“.  We  solve  (2.1) 
by  Newton  iteration  which  constructs  the  sequence  [x^}  as  follows 

(2.2)  xi+1  = - F'^)  ^(x^ 

where  x^  is  an  initial  guess  and  i = 0,1 Suppose  that  F is  sufficiently 

regular  and  a is  a simple  zero  of  (2.1),  i.e.,  F(o)  * 0 and  F' (0)  * exists  and 
is  bounded.  It  is  well-known  that  if  x^  is  sufficiently  close  to  a then  {x^} 
converges  to  a and  the  order  of  convergence  is  two,  see,  e.g.,  Kantorovich  [48], 
Ortega  and  Rheinboldt  [70]  and  Rail  [69]. 

In  this  paper  we  are  interested  in  sharp  bounds  on  how  close  x^  has  to  be 
to  a to  get  quadratic  convergence  of  Newton  iteration  and  how  close  x^  has  to 
be  to  a to  guarantee  good  complexity. 

We  begin  with  a theorem  which  describes  the  character  of  convergence  of 
Newton  iteration. 

Let  a be  a simple  zero  of  F,  = |^-a||  and  J = {x:  |^-nr||  £ r}.  Assume 
F* (x)  exists  and  is  a Lipschitz  operator  for  x 6 J.  Define 


(2.2)  A2-A2(r) 


sup  - 
x,y€J 


.'.po-r  ty>.3U 
-y  II 


r 1 

i -a 
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Note  that  if  F"  is  continuous  for  x € J then 
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A^D  = sup  |^'(»)  1 II 

x€J 

which  shows  that  in  this  case  A2  is  a bound  on  the  "normalized"  second  derivative. 
Let  q be  any  number  such  that  0 < q < 1. 

Theorem  2 . 1 

If  F'  is  a Lipschitz  operator  in  J, 

(2.3)  A2T  - 1+2q  » 

(2.4)  xQ  € J, 

then  Newton  iteration  is  well-defined  and 

(2.5)  lim  = a,  e1+1  < qet>  Vi, 

(2.6)  Vi*®,*! 

where  (1  - Ik^e^) . 

If  F is  continuously  twice-dif ferentiable  at  a then 

(2.7)  xi+1  - »^FlW‘1F"(a)(xi-(y)2+  od^-aff). 

Proof 

The  proof  is  by  induction.  Let  x^  € J and  let 

(2.8)  R(x;x  ) - *{F'(x  + t(x-x  ))  - F' (x  ) }(x-x  )dt. 

l 0 1 1 

Define  w ■ w(x;x^)  as  the  polynomial  of  first  degree  which  interpolates  F(x^) 
and  F'  (x^) . Then  w(x;xt>  - F^)  + F'fx^Hx-x^)  and  the  next  approximation 
x^  ^ is  the  zero  of  w whenever  F' (x^)  is  invertible.  From  Rail  [69,  p.124] 


we  get  the  error  formula 
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(2.9)  F (x)  - w(x;xj  = RCxjx^) 


for  x € J while  due  to  (2.2), 


(2.10)  |^"  (?)  1R(x;xi)  ||  £ A2  Ux-Xj^ \f  . 


From  the  definition  of  A2  we  have 


||l-F'  (a)  ~ r 1 (x)  ||  £ 2A2||x-Qr||  * * -jg-  < 1. 

This  means  that  F' (x)  is  invertible  for  any  x f J and  x^+^  is  well-defined. 


Furthermore 


|^'(x)‘1F'(^)||  £ 1/(1  - 2A_  | (x—  Of 1 1 ), 


see  Rail  [69,  p . 3 6 1 . We  can  rewrite  the  polynomial  w as  w(x;x.)  ■ F' (x.) (x-x.  ,) 

i i i+1 

Set  x = a in  (2.9).  Then  F'  (x^)  (x^+^-ar)  = R(r*;x^)  which  yields 


Vt  , A2r 


lK+r°n  * ^ <V  F'  <°>F'  R<®V  li 4 irtcr  s yrtr?  ei s “'i 

2 i 2 


due  to  (2.3).  This  proves  (2.5)  and  (2.6). 

If  F"(x)  is  continuous  at  a then 

R('v;xi)  = -j  F'^a)  (xi-o)2  + o(  l^-alj2)  , 
F'  (x.)  = F'  (or)  + 0(|(x.-a||  ) 

which  implies  (2.7). 

Remark  2 . 1 


Theorem  2.1  states  quadratic  convergence  of  Newton  iteration  since 
2 , 

ei+^  S Ciei  and  Ci  5 (l  - 2A2e^) . It  is  known  (see  Ortega  and  Rheinboldt 


[70,  p.319])  that  if  F'  is  not  a Lipschitz  operator  then  Newton  iteration 
does  not  possess,  in  general,  quadratic  convergence.  For  instance,  apply- 
ing Newton  iteration  to  F(x)  = x + x*  where  a = 5/3  we  get 

e1+1  = (a- l)e*/ (1  + ae®  1) . ■ 

Remark  2 . 2 

Theorem  2.1  states  how  fast  Newton  iteration  converges.  The  speed  of 
convergence  depends  on  the  value  of  q.  This  technique  based  on  q seems  to  be 
general  and  can  be  applied  for  any  iteration.  See  Traub  and  Wozniakowski  [77] 
where  the  class  of  interpolatory  iterations  1^,  n * 3,  is  considered.  B 
We  show  that  if  we  want  (2.5)  to  hold  then  (2.3)  is  sharp  for  any  value 
of  q,  0 < q < 1. 

Theorem  2 .2 

There  exists  an  entire  F such  that 

A2 f > and  eo  - r ei  > 

for  every  value  of  q € (0,1). 

Proof 

Let  F(x)  - x + x2,  x € 3*.  Then  ry  « 0 and  A2  (D  = 1 . Suppose  that 
A2T  * T > q/(l  + 2q)  and  let  xQ  =*  -F.  The  next  approximation  constructed 
by  Newton  iteration  is  equal  to  x^  = x^  - F'  (x^)  V (x^)  * F*V (1-2D  > qF  “ -qxQ» 
Hence  e^  > qe^.  B 

From  theorem  2.1  with  q - 1~  it  follows  that  Newton  iteration  converges 


whenever 
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a2t  < 1/3. 


We  show  that  there  exists  a problem  F such  that  A^  = l/ 3 and  eQ  - f imply 
that  Newton  iteration  does  not  converge. 


Theorem  2 ■ 3 

There  exists  a problem  F where  F'  is  a Lipschitz  operator  such  that  if 


A T"  = — Anri  p s T 

A2  3 na  0 


then  Newton  iteration  is  not  convergent. 


Proof 


Define 


(2.11) 


x + x for  x £ 0 


for  x a 0. 


Then  F(0)  =*  0,  F' (0)  = 1 and  F'  is  a Lipschitz  function  with  A^D  = 1.  Let 
F and  Xg  = Then  Xj  = and  x2  = — j.  Hence  Newton  iteration  cycles 

and  is  not  convergent.  ■ 

Rail  [74]  showed  that  assuming  the  existence  of  or  and  using  the  Kantorovich 
theorem  it  is  necessary  to  assume 


a2t  * = 0.15. 


4+2.  £ 


Thus  we  have  obtained  an  improvement  by  a factor  of  — j- — 4 2.3. 

We  define  optimal  convergence  number.  Let  X c 51  be  a set  of 
positive  numbers  r such  that  for  any  F with  a simple  zero  n such  that  F'  is 
a Lipschitz  operator  in  J,  A^  < r implies  that  an  iteration  converges 
whenever  |(xn-rv||  ^ 


I 
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Def inition  2 . 1 


r£  is  called  the  optimal  convergence  number  (with  respect  to 


a2)  if 


rc  = sup  X. 


From  theorem  2.1  and  theorem  2.3  we  have 


Corollary  2.1 


For  Newton  iteration 


(2.11) 


r = l/3. 
c 


Consider  the  non-linear  scalar  equation 


g(r)  s A2(nr,  r * o. 


Provided  F is  not  linear,  g(D  is  strictly  increasing,  g(0)  * 0,  g(®)  = eo. 
Let  h denote  the  inverse  function  g ^ . Then  Newton  iteration  converges 
provided  -o||  £ h(r) , r < 4. 


Definition  2.2 


R is  called  the  optimal  radius  of  the  ball  of  convergence  (with  respect 
c 


to  A2)  if 


Cor o 1 lary  2.2 


Rc  = h(rc) • 


For  Newton  iteration 


R = h(l/3) . 
c 


L 
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Remark  2.3 

Let  <P  and  i|r  be  any  two  iterations  whose  convergence  condition  is  of  the 
form  A^COi  < r.  Let  r^  (cp) , r^ C denote  the  optimal  convergence  numbers  of 
co  and  and  let  R(;(CP),  Rc('!f)  denote  the  corresponding  optimal  radii  of  the 
ball  of  convergence.  Then 

r (cp)  ^ r (ir) 
c c 

implies 

RC(=P)  «»  (*), 

Therefore  the  optimal  radii  of  convergence  of  two  iterations  can  be  ordered 
if  their  convergence  numbers  are  ordered.  H 

In  general,  Newton  iteration  converges  only  locally.  We  give  conditions 

under  which  Newtog  iteration  enjoys  a "type  of  global  convergence" . 

Let  F(x)  = ^ ~ F(i)  (oMx^-y)1  be  analytic  in  D = {x:  |(x-o||  < R]  and 
i=l 

(2.12)  it,.(o)"1F(i)(0>)||  5Ki-l 

i'. 

for  i = 2,3,...,  R o i. 

K 

One  way  to  find  K is  to  use  Cauchy's  formula 

If  (a)~1F(l)(cy)  ||  s M_ 
i'  t,  i 

where  M » sup  ' ( a)  ‘ XF  (x)  )|  . Setting  K - 

x€D 

which  yields  m/r1  £ R1  . 

Theorem  2.4 

If  F satisfies  (2.12)  then  Newton  iteration  converges  in  J 3 [x : |(x-a||  £ T") 


“(«’  R V 


-,  - ) we  get  H'R  s kr  s (KR) 


i-1 


where 


9- 


C1  . 1 
(2.13)  r =*  — and  cx  ~ 6 • 


Proof 


Since  | (F  * (ry)~  (x)  ||  £ f (l)  ( |(k-o||  ) where  f(x)  = x/ (1  - Kx)  then 


A?cnr  r-^ 

1 (1-KD  J 


C 1 

which  yields  T = ^p/ith  cj  - g*  This  proves  (2.13).  ■ 

Note  that  this  result  is  especially  interesting  if  the  domain  radius  R is 
approximately  equal  to  l/K,  say  R =>  Cj/k.  Then  r^=^R:^Rind  Newton 
iteration  enjoys  a "type  of  global  convergence  . 


3.  COMPLEXITY  OF  NEWTON  ITERATION 

In  the  previous  section  we  showed  that  the  sequence  of  errors  e^  * |(x^-a|| 
for  Newton  iteration  satisfies 

A26i 

(3-1}  6i+l  S 7-2a"2 e ^ whenever  A^  < i/3. 

We  want  to  find  x,  such  that  k is  the  smallest  index  for  which  e,  £ e'e„  where 
k k 0 

s',  0 < s'  <1,  is  a given  number.  Define  e £ «'  so  that 

(3.2)  ek  = ee0. 

The  complexity  of  Newton  iteration,  which  is  the  total  cost  of  computing  x^, 
k a 1,  is  given  by 


(3.3)  comp  = k • c 

where  c is  the  cost  of  one  Newton  step.  (See  Traub  and  Wozniakowskl  [76a] 
for  a detailed  discussion.)  Let  c(F^)  denote  the  cost  of  one  evaluation  of 
F^,  j = 0 and  1,  and  let  the  combinatory  cost  d be  the  cost  of  evaluating 
x^+^  = xi  " F'  (x^)  ^F(x^).  Then  the  cost  per  step  is  given  by 

(3.4)  c = c (F)  + c(F')  + d. 

Note  that  the  dimension  of  the  problem  N ^ +“.  For  multivariate  problems, 

N < +®,  we  can  assume  that  each  arithmetic  operation  costs  unity  and  c(F^) 
can  denote  the  total  number  of  arithmetic  operations  needed  to  evaluate  F^^ 
and  d 3 d(N)  is  the  cost  of  the  solution  of  the  linear  system 

g 

= -F(xi).  Hence  d(N)  = 0(N  ) with  P £ 3. 


F'  (xi)  <xi+rxi> 


For  the  rest  of  this  section  N £ +“.  Define 


(3.5)  f 


i+1 


Vl 

'-2A2£l' 


O’ 


0,1, 


Clearly,  the  sequence  {f ^ } is  a majorizing  sequence  of  {e^},  i.e.,  e^  £ f^, 

Vi.  Let  corap,  = k,c  where  k,  is  the  smallest  index  for  which  f,  £ *e„.  Of 
rl  1 1 k 0 

course,  comp  5 corap^.  Let  p = A2^o  * A2e0*  Note  that  comp^  ■ comp^(c,p) . 

We  derive  some  properties  of  comp^  as  p approaches  l/ 3 (which  is  the  necessary 
and  sufficient  condition  for  guaranteeing  convergence). 


Lemma  3. 1 

Let  n be  any  fixed  integer.  If  p 53  - 6 with  6 £ then 

4 

(3.6)  f “ (1  - g^fg  where  0 £ gj^  ^ 4i+1  - 4 


for  i = 0,1, ... , n. 


Proof 


Assume  by  induction  that  f^  = (1  - g^fg  with  0 £ g^  ^4 


i+1 


4.  Since 


fi+i  5 fo  then  8i+i  s 0 and 


Pd-8^)  fQ 
:i+l  = l-2p(l-g1«) 


which  yields  after  algebraic  manipulations 


2 i+2  2 i+2 

8i+l  * 48t  + 9 + 3^i^>  * * " 4 + 3(Si  6)  ’^4  -4, 


This  proves  (3.6). 


We  are  ready  to  prove 


Lemma  3.2 


(3.7)  ^ lg(-|)(l  + o(D)  s:  comp^e.p)  £ c log(|)(l  + o(l)) 

where  p ■ y - 6 with  6 -»  0+  and  lg  is  the  logarithm  to  base  2 . ■ 

Proof 

Recall  that  comp^  = k^c  and  we  seek  bounds  on  k^.  Since 


This  proves  the  righthand  side  of  (3.7). 

Let  6 = l/A1*^.  From  Lemma  3.1  for  i = 1,2,...,  Lri/2 J we  get 

fj,  s (1  - (4i+1-4)/4n+1)f0  * (l-2_n)  f 0 > efQ 

for  large  n.  Thus  k^  2 ln/2J  =>  ^ lg(-|)  (1  + o(l))  which  proves  the  lefthand 
side  of  (3.7).  ■ 

Lemma  3.2  states  the  important  difference  between  convergence  and  complex- 
ity of  the  sequence  {^3.  p < y assures  convergence  of  {f^}  but  with  only 
this  condition  complexity  can  be  arbitrarily  (logarithmically)  large. 

We  believe  results  similar  to  Lenma  3.2  also  hold  for  comp  (rather  than 
compj)  of  an  arbitrary  iteration  and  therefore  propose 

Conjecture  3.1 

For  any  superlinear  iteration  $ there  exists  a function  F with  a simple 
zero  n and  a number  T,  0 < T < such  that 


then  for^  « fixed  comp(xQ,e)  tends  logarithmically  to  infinity  as  AjF  approaches 
1/3  and  xQ  * F. 

Proof 

Note  that  or  * 0 and  AjCO  = 1,  see  (2.11).  Applying  Newton  Iteration  to 
(3.8)  we  get 


2 2 

-sign(xt)xi  e^ 

Xi+1  l-2sign(x1)xi  and  ei+l  = l-2ei 
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comp(xn,e) 

7(l+o(l)>  ^ T s l+o <l>  as  5 - °* 

4 c l«(i) 

This  proves  theorem  3.1. 

This  phenomenon  does  not  occur  in  the  Kantorovich  theory.  If  h ■ ^ (in 
the  usual  notation;  see  Rail  [74]),  Newton  iteration  converges  linearly  and 
the  complexity  is  always  bounded  for  fixed  e.  Furthermore  the  complexity 
depends  on  lg(l/«)  rather  than  lglg(l/e)  only  if  a is  not  a simple  zero. 

Note  that  F defined  in  (3.8)  is  not  twice-dif ferentiable  at  a.  It 
seems  to  us  that  comp  can  tend  to  infinity  as  A^r  approaches  l/ 3 for  twice- 
dif  ferentiable  problems.  Therefore  we  propose 

Conjecture  3.2 

There  exists  a twice-dif ferentiable  problem  F with  a simple  zero  a such 
that  the  comp  = comp(A20  of  Newton  iteration  satisfies 

lim  comp(A.r)  = +°°  | 

A2^l/3" 

Theorem  3.1  states  that  if  A2T  is  close  to  l/ 3 then  complexity  can  be 
arbitrarily  large.  We  seek  a bound  on  A^  (stronger  than  A2F  < l/ 3)  to  be 
sure  that  complexity  is  not  too  large.  Define 

(3.9)  t = max ( 1 , lg  l/e) . 

Theorem  3.2 

If  A2 T £ l/4  then  the  complexity  of  Newton  iteration  satisfies 


(3.10)  comp  Sc  lg(l+t). 


Proof 


Let  h1+1  - Kh*  with  K ■ k^l  (1  - 2A2eQ)  and  hQ  =>  eQ.  From  (3.1),  e±  shj, 
Vi.  In  Traub  and  Wozniakowski  [76a]  we  proved  that  if  w * (Ke^)"*  2 2 then 
the  complexity  comp(h)  of  the  sequence  [h^)  is  bounded  by 


comp (h)  Sc  lg(l+t) 


Since  comp  Scomp(h),  (3.10)  follows. 


Remark  3.1 

We  have  chosen  the  endpoint  w = 2 in  the  inequality  w ^ 2 only  for  con- 
venience and  definiteness.  Any  value  w bounded  away  from  unity  can  be  used 
to  get  a good  bound  on  complexity. 


Theorem  3.2  motivates  the  definition  of  the  optimal  complexity  number  for 
a superlinear  iteration.  Let  Y c 9?  be  a set  of  positive  numbers  r such  that 
for  any  F with  a simple  zero  a such  that  F'  is  a Lipschitz  operator  in  J, 

A2T  s r implies  that  an  iteration  converges  and  comp  s c lg(l+t),  Vt  ^ 1, 
whenever  |^g-or||  s r. 

Definition  3. 1 

r is  called  the  optimal  complexity  number  [with  respect  to  A2  and  the 


complexity  criterion  comp  s c lg(l+t)]  for  a superlinear  iteration  if 


r = sup  Y. 

g 


Compare  with  the  definition  of  the  optimal  convergence  number  r^  Of 

course  r Sr  and  from  Theorem  3.2  follows  that  r 2 \/u  . 
g c g 
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Theorem  3.3 

For  Newton  Iteration 


(3.11)  r 


g 4 ‘ 


Proof 


f(x)  “ x + x . Then  F(nr)  *■  0,  F'  (or)  * 1 and  A2T  ■ T.  Newton  itera- 


tion produces  {x^  such  that 


(3.12)  x1+1 


1+2  x 


2 • 


Let  t ■ 1.  Then  comp  £ c which  means  that  ej  s J e0*  Set  xq  “ "r»  r J • 
From  (3.12)  we  get 

r2  lr  1 

el  * 1-2F  2r  “ 2*0 

which  is  equivalent  to  T s l/4.  Hence  A -T  <r  s7  and  from  theorem  3.2  we 

£m  O ^ 

conclude  r m T>  ® 

g 4 


* 

I 


Remark  3.2 


It  follows  from  Remark  3.1  that  as  long  as  A2F  is  bounded  away  from  l/3 
we  are  assured  of  good  complexity.  0 


We  summarize  our  results  on  optimal  convergence  number  and  optimal  com- 
plexity number  (Corollary  2.1  and  Theorem  3.3): 

(i)  A2eQ  < rc  - 1/3  assures  convergence  but  complexity  can  be  arbitrarily 
large 


(ii)  A2e0  s r ■ l/4  assures  convergence  and  good  complexity. 
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Definition  3.2 


is  called  the  optimal  radius  of  the  ball  of  good  complexity  fwith 


respect  to  and  comp  £ c lg(l+t) ] for  a super linear  iteration  if 


R = h(r  ) 
g g 


Corollary  3.1 

For  Newton  iteration 


R = h(l/4) . 
g 


Remark  3 . 3 


A remark  analogous  to  Remark  2.3  holds  here  concerning  the  ordering  of 
the  optimal  radii  of  good  complexity  for  two  iterations.  I 


The  ratio  between  the  optimal  radius  of  the  ball  of  convergence  and  the 
optimal  radius  of  the  ball  of  good  complexity. 


_£  = Hi/3) 
R h(  1/4) 


g 


indicates  what  stronger  condition  must  be  imposed  to  assure  good  complexity. 

We  end  this  paper  by  deriving  a lower  bound  on  the  complexity  of  Newton 
iteration.  Let  F be  any  operator  for  which 


(3.!3)  el+1  s 


for  a positive  a^.  We  show  that  the  class  of  such  operators  is  not  empty. 


Let 


F (x)  = — - a,  a > 0,  x ^ 0. 


Then 


ei+l  = aei‘ 
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Recall  that  c = c(F)  + c(F')  + d is  the  cost  of  one  Newton  step  where  d 
Is  the  cost  of  solving  a linear  system  of  size  N.  Let  c^  and  c^  denote  lower 
and  upper  bounds  on  c.  If  a2eQ  2 l/t,  then  Theorem  3.1  in  Traub  and 
Wozniakowski  [76a]  yields  comp  2 cL(lgt-lglgt) . 

We  summarize  the  complexity  analysis  for  Newton  iteration. 

Theorem  3.4 

If  1 1 

Vo  5 4 and  VO  *~t 

then  the  complexity  of  Newton  iteration  satisfies 

cL(lgt-lglgt)  £ comp  5 cy  lg(l+tl 
where  t ■ max(l,  lg  l/ «) . 
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